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MyTeachingPartner’’ Pre-K 


This intervention report presents findings from a systematic review of MyTeachingPartner™ Pre-K conducted 
using the WWC Procedures and Standards Handbook (version 3.0) and the Teacher Training, Evaluation, and 
Compensation review protocol (version 3.2). No studies of MyTeachingPartner™ Pre-K that fall within the scope 
of the Teacher Training, Evaluation, and Compensation review protocol meet What Works Clearinghouse (WWC) 
group design standards. Because no studies meet WWC group design standards at this time, the WWC is unable 
to draw any conclusions based on research about the effectiveness or ineffectiveness of MyTeachingPartner™ 
Pre-K. Research that meets WWC design standards is needed to determine the effectiveness or ineffectiveness 
of this intervention. 


Intervention Description’ 


MyTeachingPartner™ Pre-K is a professional development program for early education teachers that incorporates four 
resources, used individually or together, to provide teachers with the knowledge and skills to engage in more effective 
student interactions. First, a video library provides annotated examples of high-quality teacher—student interactions. 
Second, an interactive training uses the video library to practice and discuss effective teacher-student interactions. 
Third, a three-credit college course aims to improve teachers’ knowledge about interactions that improve student 
learning. Finally, a trained coach offers web-mediated coaching on classroom instruction every 2 weeks during the 
school year. The activities in MyTeachingPartner™ Pre-K are structured according to the Classroom Assessment 
Scoring System (CLASS), a teacher observation instrument that measures quality of instruction. 


Research? 


The WWC identified 11 studies of MyTeachingPartner™ Pre-K for students that were published or released 
between 1993 and 2017. 


Four studies are within the scope of the Teacher Training, Evaluation, and Compensation review protocol but do not 
meet WWC group design standards. Of these four, one study does not meet standards because it is a randomized 
controlled trial with unknown attrition and the WWC could not assess baseline equivalence.® The three remaining 
studies that do not meet WWC group design standards included only one eligible outcome, which was overaligned 
with the intervention (that is, it was too closely tailored to the intervention).* 


Five studies are out of the scope of the Teacher Training, Evaluation, and Compensation review protocol because 
they have an ineligible study design. 


Two studies are out of the scope of the Teacher Training, Evaluation, and Compensation review protocol because 
they examine outcomes that are not eligible under the review protocol. 
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Endnotes 


1 The descriptive information for this intervention comes from the developer's (http://curry.virginia.edu/) and distributor’s 
(http://teachstone.com) websites (accessed June 2017). The What Works Clearinghouse (WWC) requests developers to review 
the intervention description sections for accuracy from their perspective. The WWC provided the developer with the intervention 
description in June 2017, and the WWC incorporated feedback from the developer. Further verification of the accuracy of the 
descriptive information for this intervention is beyond the scope of this review. 


? The literature search reflects documents publicly available by March 2017. Reviews of studies in this report used the standards from 
the WWC Procedures and Standards Handbook (version 3.0) and the Teacher Training, Evaluation, and Compensation review protocol 
(version 3.2). The evidence presented in this report is based on available research. Findings and conclusions could change as new 
research becomes available. 

3 Downer et al. (2011) is a cluster randomized controlled trial in which the WWC could not assess the levels of attrition. Thus, the study 
is required to establish equivalence of the intervention and comparison groups at baseline. The authors only provided a sample that 
included imputed baseline data which, per WWC standards, cannot be used to establish baseline equivalence. 

4 These three studies—Early et al. (2017), Pianta et al. (2014), and Hamre et al. (2012)—reported the scores from the CLASS 
observational tool as their main and only eligible outcomes. Because MyTeachingPartner™ Pre-K uses the CLASS framework to 
define and structure the intervention for effective teaching—including through instructional coaching offered every 2 weeks that 
focuses on specific CLASS dimensions—WWC determined that this outcome measure was too closely tailored to the intervention 
and might not accurately measure the effect of the intervention. The intervention group teachers would have gained familiarity through 
repetition of the videotaped sessions and knowledge of what a CLASS observer would be looking for in scoring the sessions. 


Recommended Citation 


What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2018, February). 
Teacher Training, Evaluation, and Compensation intervention report: MyTeachingPartner™ Pre-K. 
Retrieved from https://whatworks.ed.gov 
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Glossary of Terms 


Attrition 


Baseline 


Clustering adjustment 


Confounding factor 


Design 


Effect size 
Eligibility 


Equivalence 


Attrition occurs when an outcome variable is not available for all subjects initially assigned to 
the intervention and comparison groups. If a randomized controlled trial (RCT) or regression 
discontinuity design (RDD) study has high levels of attrition, the validity of the study results 
can be called into question. An RCT with high attrition cannot receive the highest rating of 
Meets WWC Group Design Standards without Reservations, but can receive a rating of Meets 
WWC Group Design Standards with Reservations if it establishes baseline equivalence of the 
analytic sample. Similarly, the highest rating an RDD with high attrition can receive is Meets 
WWC RDD Standards with Reservations. 


For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and individuals 
leave the group. If a single-case design does not meet minimum requirements for phases and 
data points within phases, the study cannot receive the highest rating of Meets WWC Pilot 
Single-Case Design Standards without Reservations. 


A point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at 
baseline. In a single-case design experiment, the baseline condition is a period during which 
participants are not receiving the intervention. 


An adjustment to the statistical significance of a finding when the units of assignment 

and analysis differ. When random assignment is carried out at the cluster level, outcomes 
for individual units within the same clusters may be correlated. When the analysis is 
conducted at the individual level rather than the cluster level, there is a mismatch between 
the unit of assignment and the unit of analysis, and this correlation must be accounted for 
when assessing the statistical significance of an impact estimate. If the correlation is not 
accounted for in a mismatched analysis, the study may be too likely to report statistically 
significant findings. To fairly assess an intervention’s effects, in cases where study authors 
have not corrected for the clustering, the WWC applies an adjustment for clustering when 
reporting statistical significance. 


A confounding factor is a component of a study that is completely aligned with one of the study 
conditions, making it impossible to separate how much of the observed effect was due to the 
intervention and how much was due to the factor. 


The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed 
repeatedly within and across different phases that are defined by the presence or absence of 
an intervention (single-case design). Designs eligible for WWC review are randomized controlled 
trials, quasi-experimental designs, regression discontinuity designs, and single-case designs. 


The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 


A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 


A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Glossary of Terms 


Extent of evidence 


Gain scores 


Group design 


Improvement index 


Intervention 


Intervention report 


Multiple comparison 
adjustment 


Outcome domain 


An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence 
categories: small and medium to large. 


e small: includes only one study, or one school, or findings based on a total sample size 
of less than 350 students and 14 classrooms (assuming 25 students in a class) 


e medium to large: includes more than one study, more than one school, and findings 
based on a total sample of at least 350 students or 14 classrooms 


The result of subtracting the pretest from the posttest for each individual in the sample. Some 
studies analyze gain scores instead of the unadjusted outcome measure as a method of 
accounting for the baseline measure when estimating the effect of an intervention. The WWC 
reviews and reports findings from analyses of gain scores, but gain scores do not satisfy the 
WWC’s requirement for a statistical adjustment under the baseline equivalence requirement. 
This means that a study that must satisfy the baseline equivalence requirement and has 
baseline differences between 0.05 and 0.25 standard deviations Does Not Meet WWC 

Group Design Standards if the study’s only adjustment for the baseline measure was in the 
construction of the gain score. 


A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 


Along a percentile distribution of individuals, the improvement index represents the gain or 
loss of the average individual due to the intervention. As the average individual starts at the 
50th percentile, the measure ranges from —50 to +50. 


An educational program, product, practice, or policy aimed at improving student outcomes. 


A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an intervention, 
reviews each against design standards, and summarizes the findings of those that meet WWC 
design standards. 


An adjustment to the statistical significance of results to account for multiple comparisons 
in a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 
the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 


A group of closely-related outcomes. A domain is the organizing construct for a set of related 
outcomes through which studies claim effectiveness. 
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Glossary of Terms 


Quasi-experimental 
design (QED) 


Randomized controlled 
trial (RCT) 


Rating of effectiveness 


Regression 
discontinuity design 
(RDD) 


Single-case design 


Standard deviation 


Statistical significance 


Study rating 


Substantively important 


Systematic review 


A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 


A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 


For group design research, the WWC rates the effectiveness of an intervention in each 
domain based on the quality of the research design and the magnitude, statistical 
significance, and consistency in findings. For single-case design research, the WWC rates 
the effectiveness of an intervention in each domain based on the quality of the research 
design and the consistency of demonstrated effects. 


A design in which groups are created using a continuous scoring rule. For example, students 
may be assigned to a summer school program if they score below a preset point on a 
standardized test, or schools may be awarded a grant based on their score on an application. A 
regression line or curve is estimated for the intervention group and similarly for the comparison 
group, and an effect occurs if there is a discontinuity in the two regression lines at the cutoff. 


A research approach in which an outcome variable is measured repeatedly within and across 
different conditions that are defined by the presence or absence of an intervention. 


The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in the 
sample tend to be spread out over a large range of values. 


Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelinood that the difference is due to chance is less than 5% (p < .05). 


The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets WWC Design Standards without Reservations, Meets WWC Design Standards with 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 


A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


A review of existing literature on a topic that is identified and reviewed using explicit methods. 
A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 

the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their findings; 
4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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ZB 


Intervention Practice Quick Single Study 
Report Guide Review Review 


An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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